1 Basics

1.1 Commands

1.1.1 Sequence of number

for i in $(seq 1 20); do
  it_no=$((50*$i))
  sed -i "${line_no}ait_no = ${it_no}" $py_fname
  sed -i "${line_no}d" $py_fname
  echo "Running it = "$it_no
  pvpython $py_fname¶
done¶

1.2 String

1.2.1 Check if string is empty/null or space

str=""
if [ "$str" == "" ];then
  echo empty
elif [ "$str" == " " ];then
  echo space
elif [ ! "$str" ];then
  echo NULL
fi

if [ ! "$str" ];then
  echo NULL
fi
$ ./test.sh
empty
NULL

1.2.2 Check if substring contained in a string

#!/bin/bash
STR='GNU/Linux is an operating system'
SUB='Linux'
if [[ "$STR" == *"$SUB"* ]]; then
  echo "It's there."
fi

2 Advanced

2.1 String

2.1.1 Remove the First Character in a String

Sometimes I need to decrease the heading level in a Markdown file. For example, the following markdown texts

## Heading 1

A sentence.

### Heading 1.1

Another sentence

need to be converted to be as follows,

# Heading 1

A sentence.

## Heading 1.1

Another sentence

By using sed, we could finish this task by

sed -i 's/^#//g' file.md

2.1.2 Compare dates

todate=$(date -d 2013-07-18 +%s)
cond=$(date -d 2014-08-19 +%s)

if [ $todate -ge $cond ];
then
    break
fi

2.1.3 Get unique value from array

ids=(aa ab aa ac aa ad) && echo "${ids[@]}" | tr ' ' '\n' | sort -u | tr '\n' ' '

2.1.4 Generate random number

Generating random number between 1 and 10

$(( ( RANDOM % 10 )  + 1 ))

2.1.5 Extract number from string

echo "ref12345678"|grep -o '[0-9]\+'

echo "ref12345678"| awk -F'[^0-9]*' '{print $2}'

2.1.6 Remove first and last quote from a variable

temp="${opt%\"}"
temp="${temp#\"}"
echo "$temp"

${opt%\"} will remove the suffix " (escaped with a backslash to prevent shell interpretation).

${temp#\"} will remove the prefix " (escaped with a backslash to prevent shell interpretation).

2.1.7 Replace some chars with another char

I have a string like AxxBCyyyDEFzzLMN and I want to replace all the occurrences of x, y, z with _.

echo "$string" | tr xyz _

would replace each occurrence of x, y, z with _, giving A__BC___DEF__LMN in your example.

echo "$string" | sed -r 's/[xyz]+/_/g'

would replace repeating occurrences of x, y, z with a single _, giving A_BC_DEF_LMN in your example.

2.1.8 Replace the whole line containing a string

$ cat test
asdbl erwbniu
assdbl ebniu
asdbl1 erwbni
3asdbl erwniu
# Replace all occurences of asdbl in the text file with NEWTC
$ sed 's/asdbl/NEWTC/g' test
NEWTC erwbniu
assdbl ebniu
NEWTC1 erwbni
3NEWTC erwniu
# Replace all lines containing asdbl in the text file with NEWTC
$ sed 's/.*asdbl.*/NEWTC/g' test
NEWTC
assdbl ebniu
NEWTC
NEWTC
# Exact match of asdbl
$ sed 's/\<asdbl\>/NEWTC/g' test
NEWTC erwbniu
assdbl ebniu
asdbl1 erwbni
3asdbl erwniu

Use -i to do the in-place modification.

2.1.9 List data structure

for i in 1 2 3; do
    echo "$i"
done

listVar="1 2 3"
for i in $listVar; do
    echo "$i"
done

2.1.10 Extract number from string

$ NUMBER=$(echo "I am 999 years old." | sed 's/[^0-9]*//g')
$ echo $NUMBER
999

$ STRING="I am 999 years old."
$ echo "${STRING//[!0-9]/}"
999
OR
$ echo "${STRING//[^0-9]/}"

2.1.11 Remove leading zero from string

Number sequences starting with a 0 are interpreted as octal numbers, i.e. base 8.

$ cat test.sh
var="00011"
echo "The input is $var"
printf "The number with leading zeros is defaultly treated as octals: %d\n" $var
var=$((10#$var))
printf "Convert the octals to the number of base 10: %d\n" $var
$ ./test.sh
The input is 00011
The number with leading zeros is defaultly treated as octals: 9
Convert the octals to the number of base 10: 11

2.1.12 Split filename and extension

filename=$(basename -- "$fullfile")
extension="${filename##*.}"
filename="${filename%.*}"

2.1.13 Color text

Use these ANSI escape codes:

RED='\033[0;31m'
NC='\033[0m' # No Color
printf "I ${RED}love${NC} Stack Overflow\n"
echo -e "I ${RED}love${NC} Stack Overflow"

2.1.14 Trim string

Use xargs in the pipe.

$ echo " Bash Scripting Language " | xargs
Bash Scripting Language

2.1.15 Convert a string to lower case

POSIX standard methods are as follows,

Use tr

$ echo "$a" | tr '[:upper:]' '[:lower:]'
hi all

Use awk

$ echo "$a" | awk '{print tolower($0)}'
hi all

2.1.16 Print multi-line string (with `\n`)

printf "$text_content"

2.1.17 Removing ANSI color codes from text stream

sed 's/\x1b\[[0-9;]*m//g'

\x1b (or \x1B) is the escape special character (sed does not support alternatives \e and \033)
\[ is the second character of the escape sequence
[0-9;]* is the color value(s) regex
m is the last character of the escape sequence

2.2 System

2.2.1 Get the source directory of the bash script in the script

$ cat test.sh
#!/bin/bash
echo "pwd: `pwd`"
echo "\$0: $0"
echo "basename: `basename $0`"
echo "dirname: `dirname $0`"
echo "dirname/readlink: $(dirname $(readlink -f $0))"
$ ./test.sh
pwd: /home/data/jcshi
$0: ./test.sh
basename: test.sh
dirname: .
dirname/readlink: /home/data/jcshi

readlink will resolve the script path to an absolute path from the root of the filesystem

2.2.2 Exit script

Use exit or return under different running modes ./test.sh, source ./test.sh and . ./test.sh.

./test.sh opens a sub-terminal while the sourced running mode source ./test.sh and . ./test.sh runs in the current terminal.

$ cat test.sh
echo $?
drga
ierr=$?
if [ "$ierr" = '127' ]; then
  echo "FAIL"
  exit -1
  # return
fi
echo "PASS"
$ ./test.sh
0
./test.sh: line 2: drga: command not found
FAIL
$ source ./test.sh
# This command exits the terminal
$ . ./test.sh
# This command exits the terminal

$ cat test.sh
echo $?
drga
ierr=$?
if [ "$ierr" = '127' ]; then
  echo "FAIL"
  # exit -1
  return
fi
echo "PASS"
$ ./test.sh
0
./test.sh: line 2: drga: command not found
FAIL
./test.sh: line 7: return: can only `return' from a function or sourced script
PASS
$ source ./test.sh
0
./test.sh:2: command not found: drga
FAIL
$ . ./test.sh
127
./test.sh:2: command not found: drga
FAIL

2.2.3 Automatic exit from Bash shell script on error

Use the set -e builtin:

#!/bin/bash
set -e
# Any subsequent(*) commands which fail will cause the shell script to exit immediately

Alternatively, you can pass -e on the command line:

bash -e my_script.sh

2.2.4 Redirect stderr

Use the following script as the example.

echo $?
# This command is not a valid command. Thus it throws stderr.
drga
echo $?

Redirect stdout to one file and stderr to another file:

$ ./test.sh >out 2>err
$ cat out
0
127
$ cat err
./test.sh: line 2: drga: command not found

Redirect stdout to a file (>out), and then redirect stderr to stdout (2>&1):

$ ./test.sh >out 2>&1
$ cat out
0
./test.sh: line 2: drga: command not found
127

Redirect both to a file (this isn’t supported by all shells, bash and zsh support it, for example, but sh and ksh do not):

$ ./test.sh &>out
$ cat out
0
./test.sh: line 2: drga: command not found
127

Pipe stderr instead of stdout

First redirect stderr to stdout — the pipe; then redirect stdout to /dev/null (without changing where stderr is going):

$ ./test.sh 2>&1 >/dev/null
./test.sh: line 2: drga: command not found

Note the difference between this method and the 2nd point.

For the details of I/O redirection in all its variety, see the chapter on Redirections in the Bash reference manual.

2.3 File and Directory

2.3.1 Compare 2 file modification dates

   FILE1 -nt FILE2
          FILE1 is newer (modification date) than FILE2

   FILE1 -ot FILE2
          FILE1 is older than FILE2

if [ "$source_file" -nt "$target_file" ]
then
    printf '%s\n' "$source_file is newer than $target_file"
fi

2.3.2 Find the line number of a matching pattern

line_no=$(grep -n 'it_no = ' $py_fname | awk -F':' '{print $1}')

2.3.3 Find out which directory the script file resides

# Absolute path to this script, e.g. /home/user/bin/foo.sh
SCRIPT=$(readlink -f "$0")
# Absolute path this script is in, thus /home/user/bin
SCRIPTPATH=$(dirname "$SCRIPT")
echo $SCRIPTPATH

2.3.4 Check if Command Exists in a Shell Script

if ! type "$foobar_command_name" > /dev/null; then
  # foobar_command_name does not exit
fi

2.3.5 Change Directory in a Shell Script

cd $HOME does not take effect if you do ./test.sh. You should do as follows,

source ./test.sh# or. ./test.sh

2.4 Others

2.4.1 Run `nohup` and write its pid to a file

nohup ./myprogram.sh > /dev/null 2>&1 & echo $! > run.pid

2.4.2 `-bash: warning: setlocale:`

Warning is something like

-bash: warning: setlocale: LC_CTYPE: cannot change locale (en_US.UTF-8): No such file or directory

You need to generate the file.

[root@centos6-vm01 ~]# vim /etc/environment         #添加下面两行内容
 LANG="en_US.UTF-8"
 LC_ALL=
[root@centos6-vm01 ~]# source /etc/environment
[root@centos6-vm01 ~]# localedef -v -c -i en_US -f UTF-8 en_US.UTF-8

You may need to install localedef.

2.4.3 Set bash options

shopt is a shell builtin command to set and unset (remove) various Bash shell options. To see current settings, type shopt.

To enable (set) option use the following command:

shopt -s optionNameHere

To disable (unset) option use the following command:

shopt -u optionNameHere

2.4.4 Check if the shell is interactive

if [ -z "$PS1" ]; then
        echo This shell is not interactive
else
        echo This shell is interactive
fi

Reference: 6.3.2 Is this Shell Interactive?

2.4.5 Sum integer of each one per line


# jcshi @ master in ~ [13:03:51]
$ awk '{s+=$1} END {print s}' tmp.dat
10
# jcshi @ master in ~ [13:04:25]
$ cat tmp.dat
1
2
3
4

2.4.6 Compare two floating number

Use bc program.

if (( $(echo "$num1 > $num2" |bc -l) )); then
  ...
fi

Shell Script

2023-05-06